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PRELIMINARY  EVALUATION  OF  A  NOVEL  SIMULATION-BASED  TOOL  FOR 
TRAINING  RAPID  DECISION-MAKING  SKILLS 

EXECUTIVE  SUMMARY 


Research  Requirement 

The  Army’s  Future  Force  concept  exploits  the  opportunities  made  possible  by  advances 
in  our  capacity  to  quickly  gather,  organize,  and  distribute  battlespace  information 
available  from  multiple  sensor  and  database  systems.  However,  as  systems  are  being 
planned  to  train  the  decision-making  skills  required  by  these  advanced  communication 
and  information  systems,  there  is  a  continuing  need  to  develop  the  decision-making 
competence  of  small-unit  leaders  who  operate  in  current  conventional  environments.  A 
novel  computer-based  training  tool,  called  the  Simulated  Field  Exercise  (SimFX)  tool, 
was  developed  that  takes  into  consideration  various  combinations  of  conventional  and 
more  technologically  advanced  operational  environments.  This  report  documents  the 
approach  and  the  results  of  a  preliminary  evaluation  of  the  SimFX  training  tool. 

Procedure 

The  SimFX  was  evaluated  using  the  results  from  a  questionnaire  completed  by  19 
participants  in  a  hands-on  workshop  conducted  for  a  broad  cross-section  of  trainers  and 
training  developers  at  Fort  Benning,  Georgia.  One  of  the  developers  of  SimFX 
demonstrated  a  full-mission  scenario  and  two  deliberative  practice  exercises.  Then, 
each  participant  was  encouraged  to  use  and  explore  the  properties  of  each  training 
component  of  SimFX.  The  developer  also  demonstrated  how  an  authoring  tool  could 
make  changes  to  the  full-mission  scenario  and  create  a  new  scenario  from  scratch. 

Findings 

This  preliminary  evaluation  of  the  SimFX  tool  showed  that  the  workshop  participants 
believed  it  could  be  an  effective  tool  for  training  information-processing  and  decision¬ 
making  skills  and  that  the  users  would  be  personally  involved  in  the  training  they 
received.  The  participants  also  indicated  that  SimFX  was  relatively  easy  to  use  for 
training  and  for  authoring  existing  and  new  training  scenarios  and  practice  exercises. 
Equally  important,  the  SimFX  tool  was  perceived  as  being  an  aid  for  the  development  of 
the  cognitive  skills  required  for  both  the  current  force  and  for  the  envisioned  Future 
Force  environment. 

Utilization  and  Dissemination  of  Findings 

The  results  of  this  preliminary  evaluation  of  SimFX  have  been  presented  to  senior 
leaders  of  Infantry  training  and  training  development  at  Fort  Benning.  These  key 
Infantry  training  personnel  have  been  invited  to  participate  in  a  follow-on  workshop 
designed  to  further  the  transition  of  this  research  product  into  the  pool  of  training 
systems  available  to  institutional  activities  and  operational  units. 
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Preliminary  Evaluation  of  a  Novel  Simulation-Based  Tool 
for  Training  Rapid  Decision-Making  Skills 

INTRODUCTION 

The  Infantry  Forces  Research  Unit  of  the  U.S.  Army  Research  Institute  for  the 
Behavioral  and  Social  Sciences  is  conducting  research  to  evaluate  the  training  potential 
of  desktop  simulations  of  dismounted  Infantry  operations  (Beal,  2005;  Beal  &  Christ, 
2004,  2005;  Centric,  Beal,  &  Christ,  2005).  The  desktop  simulations  evaluated  were 
developed  to  provide  Infantry  leaders  with  low-cost,  repetitive  opportunities  to 
experience  realistically  the  consequences  of  executing  an  operations  order  and  the 
challenges  inherent  in  making  hasty  changes  to  those  orders  in  response  to  emerging 
tactical  conditions.  Generally,  the  declared  purpose  of  these  simulations  was  to  train 
the  types  of  cognitive  skills  needed  by  Infantry  leaders  to  make  optimal  decisions  in  the 
current  or  contemporary  operating  environments. 

However,  in  keeping  with  the  Army’s  modernization  plan,  there  is  a  simultaneous 
need  to  develop  and  evaluate  desktop  training  tools  that  have  a  high  potential  to 
enhance  the  types  of  cognitive  skills  that  would  enable  rapid  decision  making  in  the 
proposed  future  operating  environment  (Department  of  the  Army,  2005).  The  Army’s 
Future  Force  concept  exploits  the  enormous  opportunities  made  possible  by  advances 
in  our  capacity  to  quickly  gather,  organize,  and  distribute  battlespace  information.  It  can 
be  argued  that  the  transition  from  the  current  to  the  future  operating  environments  may 
produce  changes  in  the  nature  of  the  information-processing  and  decision-making  skills 
currently  required  to  achieve  decision-making  proficiencies  (Mosier,  2001). 

Several  years  ago  the  author  developed  a  topic  statement  for  the  Small  Business 
Innovation  Research  (SBIR)  program  that  asked  for  the  development  of  a  computer- 
based  system  that  could  be  used  to  train  rapid  decision-making  skills  of  small  unit 
leaders  regardless  of  the  level  of  technology  used  in  their  operating  environment.  The 
training  tool  was  to  be  initially  developed  for  use  by  dismounted  Infantry  platoon  leaders 
but  also  have  the  capability  to  be  used  by  leaders  at  both  higher  and  lower  echelons. 
Further,  the  emphasis  of  the  tool  was  to  train  the  leader’s  ability  to  access,  integrate, 
and  effectively  use  information  from  multiple  sources  to  improve  his  decision-making 
proficiency.  Finally,  special  importance  was  placed  on  the  role  of  the  information 
provided  or  available  to  the  leader,  but  not  on  the  source  of  the  information,  the  format 
and  structure  of  the  information  (i.e.,  its  analog  or  digital  format),  or  the  means  through 
which  the  information  was  presented  (i.e.,  the  user-system  interface  or  knobology). 
Based  on  the  quality  of  the  background  work  and  plans  accomplished  during  a  phase  I 
effort,  a  phase  II  SBIR  contract  for  this  topic  was  awarded  to  Micro  Analysis  and  Design 
(MAAD),  Inc.  During  the  phase  II  effort,  MAAD  developed  a  tool,  called  the  Simulated 
Field  Exercise  or  SimFX,  that  met  the  objectives  of  the  SBIR  topic. 
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Key  Feature  of  SimFX 

The  one  feature  of  SimFX  that  distinguishes  it  from  other  desktop  simulations  that 
have  been  evaluated  at  Fort  Benning  is  that  it  uses  a  discrete,  outcome-driven 
simulation  for  training  leader  decision  making  rather  than  a  simulation  driven  by  inputs 
from  the  virtual  operating  environment.  (A  more  complete  description  of  the  concepts 
and  methods  that  underlie  the  development  of  the  SimFX  training  tool  is  provided  in 
Appendix  A.)  Unlike  most  other  desktop  simulations,  the  approach  used  to  create 
SimFX  training  scenarios  does  not  depend  on  the  development  of  an  immersive  virtual 
reality  in  which  the  user  performs  his  small  unit  leader  tasks.  Consequently,  the  SimFX 
software  does  not  need  to  model  autonomous  intelligent  adversaries,  and  it  does  not 
need  to  adjust  for  differences  in  the  simulated  versus  real  world  states  to  meet  the 
training  objectives.  These  features  characteristic  of  most  other  desktop  simulations 
make  them  hard  to  maintain  and  almost  impossible  to  adjust  for  changes  in  the  training 
tasks  and  conditions. 

The  outcome-driven  simulation  incorporated  in  SimFX  exploits  the  cognitive  realism 
that  is  created  when  the  learner  engages  in  an  interactive,  branching  storyline.  The 
SimFX  simulation  advances  the  leader  from  one  decision  point  to  the  next,  and, 
therefore,  focuses  the  leaders  on  making  decisions  using  information  at  their  disposal 
rather  than  on  their  experiencing  the  subtleties  of  a  virtual  environment.  The  developer 
of  SimFX  utilized  techniques  to  manage  the  size  of  the  decision  tree  that  drives  the 
decision-making  points  in  the  story  line.  These  techniques  permit  ground  truth  to  be 
adjusted  as  necessary  to  meet  the  training  objectives  and  permit  easy  modifications  of  a 
scenario  and  the  creation  of  new  scenarios. 

It  became  apparent  during  an  in-progress  review  meeting  for  this  SBIR  contract  in 
mid-October  2005  that  the  SimFX  tool  had  been  developed  to  the  point  that  it  should  be 
reviewed  and  evaluated  by  Infantry  trainers  and  training  developers  at  Fort  Benning. 

The  software  development  was  sufficiently  complete  to  support  both  authoring  and 
execution  of  training  scenarios.  An  in-house  evaluation  of  the  initial  beta-version  of  the 
software  established  a  proof  of  concept  with  a  simple  test  bed  scenario.  The  test  of  the 
beta  software  demonstrated  successfully  how  decisions  points  of  different  types  can  be 
implemented  to  insure  the  face  validity  of  the  story  line,  and  to  produce  an  engaging 
training  scenario.  The  developers  created  a  doctrinally  correct  set  of  four  full-mission 
scenarios  and  two  deliberative  practice  exercises.  The  latter  presented  repeatedly  a 
single  type  of  decision  task,  e.g.,  deciding  whether  two  images  of  an  environmental 
feature  were  the  same  or  different.  It  was  decided  that  a  revised  beta-version  of  the 
SimFX  software  could  be  demonstrated  during  a  hands-on  workshop.  This 
demonstration  would  provide  a  basis  for  conducting  a  preliminary  evaluation  of  its 
potential  for  training  in  the  Infantry  community. 

Purpose  of  This  Report 

This  report  documents  the  preliminary  evaluation  the  SimFX  training  tool.  The 
report  provides  a  brief  description  of  the  procedures  used  for  conducting  a  hands-on 
workshop  in  which  SimFX  was  demonstrated.  It  also  describes  the  outcome  of  the 
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workshop  based  on  an  assessment  of  the  perceptions  and  opinions  of  the  Infantry 
trainers  and  training  developers  who  participated  in  the  workshop. 

METHOD 


Participants 

To  encourage  participation  in  the  workshop,  the  author  conducted  a  series  of 
meetings  to  briefly  describe  the  SimFX  training  tool  and  the  hands-on  workshop  to  the 
leaders  of  ten  key  training  and  training  development  activities  and  units  across  Fort 
Benning.  In  addition,  the  author  provided  information  about  the  SimFX  tool  and  the 
workshop  to  his  colleagues  in  the  Infantry  Forces  Research  Unit  and  in  two  contractor 
firms.  The  contractor  firms  regularly  provide  support  for  training  and  training 
development  to  Fort  Benning  and  to  the  Infantry  Forces  Research  Unit.  Eight  of  the 
invited  Fort  Benning  activities  and  agencies,  as  well  as  the  three  civilian  organizations 
sent  representatives  to  the  workshop.  About  30  individuals  participated  in  all  or  most  of 
the  workshop.  Of  these  participants,  19  completed  the  questionnaire  that  provided  the 
data  used  in  this  report. 

Table  1  provides  a  general  breakdown  of  the  respondents  who  completed  the 
questionnaire.  It  was  possible  to  identify  the  status  and  the  activity  or  unit  represented 
by  all  but  two  of  these  respondents.  The  respondents  were  told  that  their  responses  to 
the  questionnaire  would  be  anonymous.  There  was  no  attempt  to  gather  any 
information  about  the  respondents’  experience  in  Infantry  training  or  training 
development.  However,  it  is  reasonable  to  conclude  that  most  of  the  senior 
commissioned  officers,  noncommissioned  officers  (NCOs),  and  civilian  respondents 
possessed  considerable  experience  in  these  areas. 

Table  1 .  Number  of  Respondents  by  Status  and 


Agencies  Re 

presented 

Status 

Number  of 
Respondents 

Number  of  Different 
Agencies  Represented 

Military 

Officer 

5 

3 

NCO 

6 

4 

Civilian 

6 

3 

Unknown 

2 

~ 

Note:  Two  of  the  19  respondents  elected  not  to  provide  their  names 
and  their  status  and  affiliation  could  not  be  determined. 


Procedure 

The  SimFX  workshop  was  conducted  in  a  classroom  at  Fort  Benning  in  which  each 
participant  was  able  to  use  the  SimFX  software  with  his  or  her  own  desktop  computer. 
The  workshop  began  with  an  introductory  presentation  that  lasted  about  15  minutes  by 
the  lead  member  of  the  MAAD  team.  During  this  presentation  the  operating  sets  of 
assumptions  and  principles  that  drove  the  development  of  the  SimFX  training  tool  were 
described. 
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Following  the  introductory  presentation,  the  participants  were  told  to  turn  on  their 
computers  to  access  to  the  revised  beta-version  of  SimFX.  For  the  next  fifteen  minutes 
the  lead  contractor  demonstrated  the  full-mission  scenario  and  the  two  deliberative 
practice  exercises  that  were  contained  in  the  software.  The  participants  were 
encouraged  to  ask  questions  during  this  developer-led  demonstration  of  the  revised 
beta-version  of  the  SimFX  software. 

During  the  next  60  minutes,  the  participants  were  encouraged  to  use  and  explore 
the  properties  of  each  component  of  the  training  software.  While  the  participants  used 
the  SimFX  tool,  the  lead  developer  and  three  of  his  colleagues  (as  well  as  the  author) 
answered  questions  raised  by  the  participants  and  provided  information.  At  the 
conclusion  of  this  60-minute  hands-on  demonstration,  the  participants  were  told  they 
could  take  a  short  1 5-minute  break.  (Up  to  one-half  the  participants  elected  not  to  take 
a  break  or  to  take  only  an  abbreviated  break.  These  participants  used  some  or  all  of  the 
break  time  to  continue  investigating  the  properties  of  the  SimFX  training  tool.) 

Following  the  break,  the  lead  developer  demonstrated  how  the  authoring  tool  could 
be  used  to  make  changes  to  the  full  mission  scenario  that  was  contained  in  the  beta- 
version  of  SimFX  software.  He  then  demonstrated  how  the  authoring  tool  could  be 
used  to  create  a  new,  three-node  scenario  from  scratch.  A  hand-out  was  provided  to 
the  participants  to  guide  them  through  this  application  of  the  authoring  tool.  During  and 
following  this  demonstration,  the  participants  were  encouraged  to  raise  questions  or  to 
offer  comments  to  the  developers.  The  demonstration  of  the  authoring  capability  of  the 
SimFX  tool  lasted  about  30  minutes. 

At  the  end  of  this  final  demonstration  period,  each  participant  was  given  a  copy  of 
the  SimFX  Questionnaire.  The  participants  were  encouraged  to  complete  the  SimFX 
Questionnaire  before  they  left  the  classroom.  The  author  collected  all  the 
questionnaires  that  were  completed. 

The  SimFX  Questionnaire 

The  SimFX  Questionnaire  was  developed  to  capture  the  opinions  of  the  participants 
about  their  experiences  using  the  SimFX  tool  during  the  workshop.  A  copy  of  the 
questionnaire  is  given  in  Appendix  D.  Successive  parts  of  the  questionnaire  asked  the 
respondents  to  rate  the  following: 

•  Training  value  of  the  SimFX 

•  Extent  to  which  users  of  SimFX  would  be  personally  involved  with  the  training 

•  Ease  of  use  or  the  usability  of  SimFX  for  training  and  for  editing  or  authoring  the 
training  material  presented  with  SimFX. 

Participants  also  were  asked  to  provide  written  comments  about  SimFX  in  terms  of  its 
advantages  and  disadvantages  as  a  training  tool  and  as  an  editing  or  authoring  tool. 

In  the  first  two  parts  of  the  questionnaires,  respondents  used  a  seven-point  rating 
scale  to  indicate  their  responses  to  questions  about  SimFX.  In  these  two  parts  of  the 
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questionnaire,  a  scale  value  of  1  reflected  the  most  negative  rating  for  SimFX,  a  scale 
value  of  4  a  neutral  or  indifferent  rating,  and  a  scale  value  of  7  the  most  positive  rating 
for  SimFX.  In  part  three  of  the  questionnaire,  respondents  used  a  five-point  rating  scale 
to  indicate  their  reactions  to  positive  statements  about  the  usability  of  SimFX.  The  five 
scale  values  were  verbally  labeled  Strongly  disagree,  Disagree,  Neither  agree  nor 
disagree,  Agree,  and  Strongly  agree,  and  assigned  numeric  values  of  1  through  5, 
respectively.  Many  of  the  items  to  be  rated  in  the  questionnaire  were  modified  from 
those  used  to  assess  the  training  effectiveness  of  other  Army  simulation-based  training 
tools  (Beal  &  Christ,  2004,  2005). 

In  the  last  part  of  the  questionnaire  the  respondents  were  asked  to  describe  their 
overall  opinions  of  the  training  and  authoring  capabilities  of  SimFX.  They  were  asked  to 
describe  up  to  three  positive  and  three  negative  features  of  SimFX  for  training  and  up  to 
two  positive  and  two  negative  features  of  SimFX  for  authoring  training  scenarios  and 
exercises.  Observations  of  and  oral  comments  by  the  participants  during  the  workshop 
were  not  formally  recorded  and  are  not  presented  in  this  report. 

RESULTS 

Rated  Opinions  About  SimFX 

Training  value  of  the  SimFX  tool.  The  participants  indicated  that  they  believed 
the  SimFX  would  have  training  value.  Across  the  eleven  items  in  this  part  of  the 
questionnaire,  between  68  and  95  percent  of  the  respondents  used  one  of  the  three 
highest  rating  categories  to  indicate  a  positive  opinion  of  the  training  value  of  the 
SimFX.  (Details  of  the  ratings  for  training  value  are  presented  in  Appendix  Table  C-1). 
The  five  items  on  which  over  half  of  the  respondents  used  one  of  two  most  positive 
rating  categories  to  indicate  their  opinion  of  the  training  value  of  the  SimFX  tool  are 
listed  below  (in  descending  order  of  the  percentage  of  respondents  using  one  of  these 
two  rating  categories). 

•  Would  training  with  SimFX  help  small  unit  leaders  focus  on  critical  factors  that 
influence  tactical  decisions? 

•  Would  using  SimFX  provide  a  Soldier  opportunity  to  practice  making  sound 
tactical  decisions? 

•  Would  training  with  SimFX  be  a  valuable  learning  experience? 

•  Would  training  with  SimFX  improve  a  Soldier’s  ability  to  make  more  rapid  tactical 
decisions? 

•  To  what  extent  will  training  with  SimFX  help  a  Soldier  to  make  sound  tactical 
decisions? 

User  involvement  during  SimFX  training.  The  participants  indicated  that  users 
would  become  personally  involved  while  training  with  the  SimFX  tool.  Across  the  five 
questionnaire  items  that  assessed  this  factor,  between  52  and  95  percent  of  the 
participants  used  one  of  the  three  highest  rating  categories  to  indicate  that  a  user  would 
become  involved  with  the  SimFX  training.  (Details  of  the  involvement  ratings  are 
presented  in  Appendix  Table  C-2).  The  two  items  on  which  over  half  the  respondents 
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used  one  of  two  highest  rating  categories  to  indicate  their  belief  that  users  would  be 
personally  involved  during  SimFX  training  are  shown  below  (in  descending  order  of  the 
percentage  of  respondents  using  one  of  these  two  rating  categories). 

•  How  involved  were  you  in  the  decision-making  experiences  provided  by  SimFX? 

•  How  often  might  a  Soldier  training  with  SimFX  be  completely  focused  on  the 

decision-making  task? 

Usability  of  SimFX  as  a  training  tool.  The  participants  indicated  that  SimFX 
would  be  relatively  easy  to  use  and  that  they  would  use  it  as  a  training  tool.  Across  the 
19  items  used  to  assess  this  aspect  of  the  SimFX  tool,  between  61  and  100  percent  of 
the  respondents  either  agreed  or  strongly  agreed  with  positive  statements  about  the 
usability  of  SimFX  for  training.  (Details  of  the  ratings  of  SimFX  usability  for  training  are 
presented  in  Appendix  Table  C-3).  The  four  items  on  which  over  a  third  of  the 
respondents  used  the  strongly  agree  rating  category  for  the  usability  of  SimFX  for 
training  are  listed  below  (in  descending  order  of  the  percentage  of  respondents  using 
this  highest  rating  category). 

•  If  it  were  available,  I  would  use  SimFX  as  a  training  tool. 

•  If  the  SimFX  tool  were  available  I  would  use  it  to  train  my  Soldiers. 

•  The  "clock”  shown  in  the  SimFX  display  to  indicate  how  much  time  remains  to 
complete  a  mission  was  easy  to  understand. 

•  The  steps  required  to  use  SimFX  for  training  purposes  are  easy  to  perform. 

Usability  of  SimFX  for  authoring  training  scenarios  and  exercises.  The 

participants  indicated  they  believed  the  SimFX  software  would  make  it  relatively  easy  to 
modify  existing  training  exercises  and  to  create  new  ones  more  to  their  liking.  They 
further  indicated  that  if  the  software  were  available  to  them,  they  would  exercise  this 
capability.  Across  the  6  items  in  this  part  of  the  questionnaire,  between  61  and  94 
percent  of  the  respondents  either  agreed  or  strongly  agreed  with  positive  statements 
I  about  the  usability  of  SimFX  for  training.  (Details  of  the  ratings  for  the  usability  of  SimFX 

I  authoring  capabilities  are  presented  in  Appendix  Table  C-4).  The  three  items  on  which 

[  over  a  third  of  the  respondents  used  the  most  positive  rating  category  for  the  authoring 

capability  of  SimFX  are  listed  below  (in  descending  order  of  the  percentage  of 
respondents  using  the  strongly  agree  rating  category). 

•  If  it  were  available,  I  would  personally  create  some  new  SimFX  scenarios  or 

t  exercises. 

[  •  If  it  were  available,  I  would  personally  modify  existing  SimFX  scenarios  or 

I  exercises. 

•  With  just  a  little  practice  I  could  become  very  good  at  modifying  and  creating 
SimFX  scenarios  or  exercises. 
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Written  Comments  About  SimFX 

Seventeen  of  the  19  respondents  provided  101  written  comments  about  positive 
and  negative  features  of  the  SimFX  tool.  Sixteen  respondents  gave  36  positive 
comments  and  15  gave  28  negative  comments  about  SimFX  training  capabilities. 
Sixteen  respondents  gave  24  positive  comments  and  9  gave  1 3  negative  comments 
about  the  authoring  capability  of  SimFX.  These  written  comments  are  provided  in 
Appendix  Tables  C5-C8.  The  author’s  attempt  to  organize  these  written  comments  into 
meaningful  clusters  is  reflected  in  these  four  appendix  tables  and  is  summarized  in  this 
section  of  the  report. 

Written  comments  about  using  SimFX  for  training.  The  respondents’  positive 
comments  about  SimFX  training  followed  a  pattern  similar  to  their  rated  opinions.  The 
majority  of  the  positive  comments  referred  to  the  ease  of  using  SimFX  for  training  and 
its  use  specifically  to  train  decision-making  skills.  Another  cluster  of  the  positive 
comments  spoke  specifically  about  the  potential  use  of  SimFX  scenarios  to 
accommodate  other  current  and  emerging  training  requirements. 

Some  of  the  negative  comments  about  training  with  SimFX  were  concerned  with 
details  specific  to  the  scenario  that  was  demonstrated  in  the  workshop.  For  example, 
remote  and  robotic  sensor  systems  played  a  key  role  in  the  demonstrated  scenario  but 
most  of  the  workshop  participants  were  not  familiar  with  the  capabilities  of  these 
emerging  technologies.  However,  most  of  the  negative  comments  about  using  SimFX 
for  training  highlighted  issues  that  are  common  to  the  use  of  many  training  tools,  such 
as  the  difficulty  inherent  in  developing  good  scenarios  and  conditions  for  training,  the 
difficulty  of  getting  computers  to  use  for  training,  and  the  preference  for  live  training. 
Other  negative  comments  showed  that  some  features  of  SimFX  were  not  clearly 
demonstrated  during  the  workshop. 

Written  comments  about  the  SimFX  authoring  capability.  Positive  comments 
about  the  authoring  capability  of  SimFX  reinforced  the  rated  opinions  given  for  its 
usability.  Most  of  the  positive  comments  spoke  to  the  relative  ease  of  use  and  the 
flexibility  provided  by  the  authoring  software.  Many  respondents  indicated  they  would 
be  able  to  tailor  SimFX  scenarios  to  fit  their  specific  training  requirements.  Several 
respondents  pointed  specifically  to  the  use  of  this  capability  to  convert  currently  used 
paper-and-pencil-based  decision  exercises  into  SimFX  scenarios  and  to  use  them  in 
SimFX  to  enhance  the  training  of  tactical  decision-making  skills. 

Most  of  the  negative  comments  about  SimFX  authoring  capability  addressed 
perceived  difficulties  in  learning  how  to  author  scenarios.  Another  cluster  of  negative 
comments  were  concerned  with  whether  the  authoring  capability  could  be  used  to 
create  scenarios  for  the  current  force  that  does  not  have  access  to  the  remote  sensor 
technology  that  was  highlighted  in  the  demonstrated  scenario. 
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DISCUSSION 


The  results  showed  clearly  that  the  participants  in  the  hands-on  SimFX  workshop 
thought  the  SimFX  tool  has  the  potential  to  effectively  train  decision-making  skills.  Most 
respondents  gave  very  high  ratings  and  wrote  positive  comments  about  the  value  and 
ease  of  using  SimFX  for  training.  Over  half  the  respondents  indicated  that  Soldiers 
would  become  involved  in  the  training  provided  by  the  SimFX  tool.  Further,  most 
respondents  indicated  that,  if  SimFX  were  available,  they  would  use  it  to  train  decision¬ 
making  skills.  Some  respondents  specifically  recommended  that  currently  used  paper- 
and-pencil  decision-making  exercises  be  converted  into  SimFX  scenarios. 

While  the  rated  opinions  of  workshop  participants  were  quite  positive  toward  the 
training  value  of  the  SimFX  tool,  their  written  comments  indicated  also  some  perceived 
negative  aspects  of  training  with  the  SimFX  tool.  Many  of  these  negative  comments 
about  training  with  SimFX  were  keyed  to  only  the  specific  scenario  that  was 
demonstrated  in  the  workshop.  These  perceived  negative  aspects  of  the  SimFX 
scenario  could  be  neutralized  or  reversed  if  the  participants  had  been  exposed  to  other 
training  scenarios  that  did  not  employ  remote  and  robotic  sensors.  The  other  problems 
noted  by  the  respondents  were  generally  applicable  to  all  training  tools  and  simulations 
(e.g.,  difficulty  in  preparing  good  training  scenarios)  or  are  a  reflection  of  more 
widespread  problems  in  acquiring  training  resources  (e.g.,  scarcity  of  computers  for 
training  purposes). 

The  rated  and  written  opinions  of  the  workshop  participants  about  the  authoring 
capability  built  into  the  SimFX  tool  were  more  equivocal  than  those  about  its  training 
capability.  On  the  one  hand,  the  respondents  indicated  that  with  a  little  practice  they 
thought  they  would  be  able  to  use  the  authoring  capability  to  modify  existing  scenarios 
as  well  as  to  create  new  scenarios  to  meet  their  training  responsibilities.  However, 
some  respondents  also  indicated  that  it  might  be  difficult  to  use  the  authoring  capability 
of  SimFX.  One  possible  reason  for  the  more  ambivalent  opinion  about  the  authoring 
capability  than  the  training  capability  of  SimFX  is  that  participants  had  hands-on 
experience  using  the  training  aspects  of  the  SimFX  tool  but  received  only  a  description 
of  its  authoring  capability.  It  is  anticipated  that  planned  future  hands-on  demonstration 
of  both  the  training  and  the  authoring  capabilities  of  the  final  version  of  the  SimFX  tool 
will  reinforce  the  positive  user  opinions  of  the  tool  and  reduce  if  not  eliminate  any 
perceived  problems  with  its  use. 

CONCLUSION 

The  findings  of  this  preliminary  evaluation  of  the  SimFX  tool  suggest  that  the  tool 
may  be  both  an  effective  and  an  efficient  means  to  train  information-processing  and 
decision-making  skills.  Equally  important,  the  SimFX  tool  has  the  capability  to  aid  to  the 
development  of  the  cognitive  skills  required  for  both  the  current  force  and  for  the 
envisioned  future  force  environment.  Given  the  relatively  low  level  of  development  of 
the  SimFX  tool  used  in  this  evaluation,  the  short  duration  of  the  workshop  used  for  this 
evaluation,  and,  most  importantly,  the  absence  of  a  hands-on  demonstration  of  its 
authoring  capability,  it  is  important  to  conduct  appropriate  follow-on  evaluations  of  this 
desktop  training  simulation  as  it  is  further  refined  and  developed. 
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Appendix  A.  A  description  of  the  SimFX  training  Tool 

SIMULATION  BASED  TRAINING  FOR  THE  FUTURE  FORCE  WARRIOR  * 

Walter  Warwick,  Rick  Archer,  Alan  Brockett,  and  Patty  McDermott 
Micro  Analysis  and  Design 
Boulder,  CO 

Abstract:  In  this  paper  we  describe  techniques  we  have  adopted  to  develop  a 
computer-based,  outcome-driven  simulator  to  train  digital  information  skills  for 
small  unit  leaders  of  the  Army’s  Future  Force  Warrior  program.  We  begin  by 
contrasting  attempts  to  engender  “virtual  realism”  in  simulation  based  training 
against  attempts  to  engender  cognitive  realism  by  way  of  the  branching  storylines 
at  the  heart  of  an  outcome-driven  simulation.  We  next  present  an  example  of  how 
such  an  approach  might  be  applied  to  train  digital  information  skills  before 
turning  to  a  more  general  discussion  of  the  problems  that  such  an  approach 
entails — namely,  crafting  an  engaging  story  while  minimizing  the  combinatorial 
explosion  in  a  branching  storyline.  We  describe  how  we  have  dealt  with  these 
problems  both  by  streamlining  storylines  and  by  decoupling  student  input  from 
the  branching  process.  Finally,  we  allude  to  a  software  tool  we  have  created  that 
allows  the  training  developer  to  author  and  execute  such  outcome-driven 
simulations. 

INTRODUCTION 

Realism  is  an  essential  component  of  simulation-based  training.  For  many  computer-based 
simulations,  this  realism  is  accomplished  with  the  construction  of  a  highly  detailed,  carefully 
rendered,  synthetic  environment  coupled  with  some  sort  of  input  device  that  allows  the  student  to 
interact  with  the  simulated  environment.  So-called  virtual  reality  permits  the  student  to  explore 
the  simulated  training  environment  in  real-time,  in  a  perceptual-motor  situation  similar  to  the  real 
world.  In  principle,  anything  the  student  might  do  in  the  actual  environment  could  be  done  via 
simulation  in  a  virtual  environment.  As  long  as  the  simulated  environment  reflects  the  salient 
interactions  of  the  actual  environment,  the  student  can  gain  valuable  experience  performing  tasks 
that  are  either  too  dangerous,  or  too  expensive,  or  simply  impossible  to  perform  in  the  actual 
environment.  While  effective  for  some  types  of  training,  immersion  in  a  virtual  reality  comes 
with  its  own  issues  and  significant  overhead  that  do  not  justify  its  application  in  every  training 
domain.  (In  fact,  it  is  not  clear  that  highly  realistic  synthetic  environments  provide  useful 
training  for  dismounted  Soldiers.  Cf.  Pleban  (2003),  Christ  (2004))  Outcome-driven  simulation 
has  recently  emerged  as  one  possible  alternative  (e.g.,  Gordon,  2004).  In  outcome-driven 
simulation  the  goal  is  no  longer  to  immerse  the  student  in  a  virtual  reality  but,  rather,  to  exploit 
the  cognitive  realism  that  follows  from  engaging  the  student  in  a  story  or  vignette  where  the 
student  must  make  a  series  of  decisions  that  ultimately  affects  how  the  story  plays  out. 

Outcome-driven  simulation  trades  the  continuous  environment  of  virtual  reality  for  a  set  of 
discrete  choice  points  built  into  a  narrative  structure.  By  scripting  together  a  series  of  choice 
points  in  a  branching  storyline,  the  training  developer  maintains  control  over  the  interactions 
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between  student  and  simulation — ensuring  that  the  student  will  encounter  specific  decisions  at 
specific  times  rather  than  wandering  aimlessly  or  unpredictably  through  a  virtual  environment. 
However,  crafting  the  branching  story  line  that  constitutes  an  outcome-driven  simulation  places  a 
significant  burden  on  the  developer  to  come  up  with  an  engaging  yet  tractable  scenario.  If  the 
training  developer  constructs  a  scenario  with  too  few  choice  points  he  runs  the  risk  of 
constructing  a  simulation  that  is  no  more  engaging  than  a  short  multiple  choice  exam.  At  the 
other  extreme,  if  the  developer  tries  to  string  together  too  many  choice  points  he  will  quickly 
find  himself  lost  in  a  combinatorial  explosion.  The  training  developer  must  strike  a  balance 
between  engaging  the  student  and  managing  the  complexity  of  a  scenario  maintaining  all  the 
while  some  semblance  of  continuous  flow  and  believability  throughout  the  scenario,  no  matter 
which  choices  the  trainee  makes. 

In  this  paper  we  describe  how  we  have  struck  such  a  balance  in  the  development  of  a 
computer-based,  outcome-driven  training  simulator.  The  simulator  is  intended  to  train  the  small 
unit  leaders  of  the  U.S.  Army’s  Future  Force  Warrior  (FFW)  program  to  make  sense  of  the  ever 
wider  array  of  information  technologies  available  as  the  Army  transforms  itself  to  fight  on  the 
“digital  battlefield.”  The  emphasis  on  digital  technology  and  the  ability  of  the  Soldier  to  fuse 
information  from  various  sources  makes  this  domain  especially  well  suited  to  an  outcome-driven 
simulation.  A  series  of  discrete  choice  points  can  be  used  to  present  the  Soldier  with  particular 
pieces  of  information  that  must  be  fused  to  make  the  “right”  decision  (i.e.,  the  decision  that 
allows  the  scenario  to  unfold  in  the  manner  intended  by  the  training  developer).  Still,  crafting 
the  training  scenarios  has  been  far  from  trivial.  We  describe  below  how  we  have  exploited  some 
of  the  “chapter-based”  techniques  from  Gordon  (2004)  to  manage  combinatorial  explosion,  along 
with  insights  from  the  gaming  community  that  inspired  us  to  decouple  the  overt  choices  made  by 
the  student  from  those  that  actually  dictate  the  flow  of  control  through  the  scenario.  In  this  way 
we  present  the  illusion  that  the  student  is  interacting  with  a  simulated  world  that  exists 
independently  without  incurring  the  overhead  that  would  otherwise  be  necessary  to  maintain  a 
consistent  world  state.  We  also  allude  to  a  general  software  tool  we’ve  built  that  allows  us  to 
implement  these  techniques  in  the  development  of  computer-based,  outcome-driven  training 
scenarios. 


SCENARIOS  TO  TRAIN  DIGITAL  SKILLS  FOR  THE  FFW 

Although  a  good  deal  has  been  written  recently  about  the  impacts  of  digital  technologies 
and  their  implications  for  training,  our  work  was  motivated  by  the  straightforward  observation 
that  expertise  is  generally  built  on  a  foundation  of  hands-on  experience.  So,  rather  than  focus 
training  on  the  specifications  and  capabilities  of  new  digital  technologies — the  “knobology”  of 
new  technology — we  set  out  to  provide  students  with  computer-based  scenarios  that  would  force 
them  to  resolve  ambiguous  or  contradictory  sensor  readings,  fuse  disparate  sources  of 
information,  filter  information,  manage  resources,  (e.g.,  time,  network  bandwidth)  and  learn  how 
to  employ  sensors  to  the  greatest  effect  in  a  tactical  situation. 

For  example,  at  one  decision  point  we  might  ask  the  student  to  pick  among  three  routes  to 
a  waypoint.  The  paths  are  presented  on  an  electronic  display  of  a  map.  The  student  has  the 
ability  to  query  various  information  sources.  In  addition  to  traditional  information  sources  (e.g., 
an  operations  order,  radio  communications,  map  overlays)  the  student  can  query  unattended 
acoustic  sensors,  reconnaissance  from  unmanned  air  and  ground  vehicles  and  spot  reports  from  a 
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densely  connected  communications  network.  Choosing  the  correct  path  means  querying  the 
appropriate  sensor  and  making  good  use  of  the  information  it  provides.  In  this  case,  the  decision 
was  crafted  so  that  the  student  must  recognize  that  the  indication  of  foot  traffic  reported  by  an 
unattended  acoustic  sensor  in  the  vicinity  of  one  route  is  inherently  ambiguous  and  that  the 
determination  of  whether  it  is  due  to  enemy  or  friendly  activity  along  the  route  depends  on 
querying  another  sensor — perhaps  inspecting  recent  aerial  reconnaissance.  The  choice  can  be 
further  complicated  by  layering  tactical  considerations  and  time  management  demands  (e.g.,  the 
shortest  route  offers  less  cover). 

Although  seemingly  straightforward,  implementing  this  decision  point  depended  on  the 
solutions  to  several  interrelated  questions.  First,  we  had  to  decide  how  information  would  be 
presented.  While  we  wanted  to  preserve  the  “look  and  feel”  of  the  information  sources,  we 
didn’t  want  the  student  to  become  mired  in  the  painstaking  analysis  of  a  grainy  reconnaissance 
photograph  or  the  interpretation  of  a  particular  acoustic  signature  in  a  noisy  signal.  Instead,  we 
opted  to  present  information  from  these  sources  abstractly  (usually  as  text-based  reports  from  a 
notional  intelligence  analyst  who  reviews  sensor  data),  to  emphasize  how  the  student  should 
integrate  such  facts  once  presented  rather  than  train  interpretation  of  raw  data.  More  generally, 
the  abstract  representation  reflects  the  desire  to  steer  away  from  a  detailed  underlying  model 
where  a  consistent  “world  state”  can  be  maintained  and  presented  to  the  student  (via  additional 
and  comparably  complex  sensor  models).  Instead,  the  training  developer  simply  specifies  the 
information  provided  to  the  student  at  each  decision  point,  tweaks  the  simulated  world  as 
necessary  (e.g.,  adding  enemies  at  a  location,  removing  assets),  much  in  the  same  way  that  a  so- 
called  observer-controller  will  change  the  course  of  a  live  training  exercise  to  suit  the  training 
objectives.  But  while  the  training  developer  gains  greater  control  of  the  simulation  in  this  way,  it 
comes  at  a  cost.  Without  detailed,  underlying  models  to  maintain  a  consistent  world  state,  it  falls 
on  the  training  developer  to  manage  the  complexity  and  consistency  of  the  unfolding  scenario. 

As  indicated  above,  managing  this  complexity  is  difficult. 

Even  with  only  a  few  choices  at  each  decision  point,  keeping  track  of  all  the  possible  paths 
through  a  scenario  becomes  unmanageable  after  a  handful  of  decisions.  While  some  degree  of 
combinatorial  explosion  is  inevitable,  it  can  be  minimized  in  a  number  of  ways.  First,  as  Gordon 
(2004)  describes,  a  branching  scenario  can  be  pruned  by  introducing  “chapters”  whereby  a  series 
of  decisions  ultimately  funnel  back  to  a  single  decision.  For  example,  we  ask  the  student  a  short 
series  of  questions,  each  of  which  asks  where  he’d  move  to  next  given  the  available  information 
sources  (which  change  from  decision  to  decision).  But  rather  than  ramify  the  student’s  decisions 
throughout  the  entire  scenario,  we  introduce  a  new  series  of  questions  by  discontinuously 
moving  the  student  to  a  new  location  that  could  plausibly  be  reached  no  matter  which  route  the 
student  chose  previously. 

A  second  technique  for  minimizing  combinatorial  explosion  is  simply  to  avoid  it  in  the 
first  place  by  posing  non-branching  decisions.  Such  decisions  either  ask  the  student  to  provide 
factual  responses  about  digital  technologies  (e.g.,  “Can  your  unmanned  acoustic  sensor  field  at 
the  objective  detect  truck  traffic  on  the  readjust  east  of  the  objective?”),  or  we  can  ask  the 
student  to  estimate  the  resources  required  to  execute  particular  phases  of  the  mission,  or,  finally, 
use  rhetorical  strategies  to  force  the  student  to  deepen  his  thought  about  the  tactical  situation 
(e.g.,  “Have  you  considered  how  your  operational  tempo  would  be  affected  if  your  were  flanked 
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en  route  to  the  objective?”).  While  the  student  will  be  prompted  for  a  response,  the  response 
does  not  change  how  the  scenario  advances. 

Finally,  borrowing  techniques  from  the  gaming  community,  we  have  found  it  is  possible  to 
present  the  student  with  a  seemingly  genuine  decision  (i.e.,  a  choice  that  affects  outcomes) 
without  having  to  represent  those  outcomes  in  the  scenario.  The  trick  here  is  not  to  predicate  the 
outcome  of  the  decision  on  what  the  student  actually  chooses,  but  rather,  on  what  the  student 
knew  or  should  have  known  before  he  made  his  choice  (which  we  can  infer  by  keeping  track  of 
which  information  assets  were  queried).  Much  in  the  same  way  that  a  video  game  designer  will 
program  a  monster  to  appear  in  whatever  room  the  player  enters,  we  can  ensure  that  bad  things 
will  happen  whenever  a  student  fails  to  make  the  best  use  of  the  information  assets  at  his 
disposal.  Returning  to  our  earlier  example,  independent  of  route  the  student  actually  chooses,  we 
can  place  an  enemy  ambush  whenever  the  student  fails  to  disambiguate  the  reports  from  his 
acoustic  sensors.  Conversely,  the  enemy  will  be  absent  whenever  the  appropriate  combination  of 
sensors  is  queried  and  we  will  reward  the  student  for  recognizing  the  original  ambiguity,  thus 
reinforcing  the  training  objective.  While  this  style  of  question  requires  the  training  developer  to 
specify  training  feedback  (i.e.,  outcomes)  for  a  potentially  large  number  of  sensor  combinations, 
it  allows  scenarios,  or  large  parts  of  them,  to  be  developed  without  any  branching  and  so  the 
level  of  effort  tends  to  grow  linearly  rather  than  exponentially  in  the  depth  of  the  scenario. 

DISCUSSION 

We  built  a  software  tool  that  allows  us  to  implement  all  three  styles  of  decision  points. 

The  training  developer  is  able  to  specify  the  scenario  structure  using  a  Decision  Tree  Editor 
(depicted  in  Figure  1  below).  The  Decision  Point  Editor  (depicted  in  Figure  2  below)  allows  the 
training  developer  to  specify  each  decision  point,  including  the  information  that  will  be 
available,  the  choices  the  student  can  make  and  the  feedback  the  student  will  receive. 


Figure  1  Decision  Tree  Editor 
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Fieure  2  Decision  Point  Editor 


Our  effort  to  date  has  been  on  software  development,  as  opposed  to  content  development. 
We  have  created  simplistic  scenarios  to  anchor  intuitions  and  focus  our  efforts,  but  our  real 
interest  is  in  tool  development  (although  we  have  just  begun  working  with  an  infantry  subject 
matter  expert  to  develop  more  robust  training  scenarios).  Accordingly,  our  focus  on  evaluation 
has  been  limited  to  the  usability  of  the  software  and  its  ability  to  represent  the  kinds  of 
information-rich  decision  points  we  think  might  be  important  training  points  for  the  FFW.  The 
evaluation  of  training  effectiveness  will  have  to  wait  until  richer  content  can  be  developed. 

While  our  approach  places  a  non-trivial  demand  on  the  developer  to  produce  a  well- 
crafted  outcome  driven  scenario,  the  required  effort  pales  in  comparison  to  that  required  to 
develop,  maintain  and  use  more  immersive  simulation  environments.  At  the  other  extreme,  we 
are  inspired  by  a  generation  of  paper  and  pencil  exercises,  called  Tactical  Decision  Games,  that 
the  U.S.  Marines  use  to  suspend  disbelief  and  engage  the  story  behind  the  training  scenario.  We 
see  our  approach  as  a  middle  road  between  high-fidelity  simulations  and  pencil-and-paper 
exercises. 

References 

Beal,  S.A.  &  Christ,  R.E.  (2004).  Training  Effectiveness  Evaluation  of  the  Full  Spectrum 

Command  Game  (Technical  Report  1 140).  Arlington,  VA:  U.S.  Army  Research  Institute 
for  the  Behavioral  and  Social  Sciences. 

Gordon,  A.  (2004).  Authoring  Branching  Storylines  for  Training  Applications.  Proceedings  of 
the  Sixth  International  Conference  of  the  Learning  Sciences  (ICLS-04),  Santa  Monica,  CA. 

Pleban,  R.  J.  &  Salvetti,  J.  (2003).  Using  Virtual  Environments  for  Conducting  Small  Unit 
Dismounted  Mission  Rehearsals  (Research  Report  1806).  Arlington,  VA:  U.S.  Army 
Research  Institute  for  the  Behavioral  and  Social  Sciences. 


Appendix  B.  SimFX  Questionnaire 


Simulated  Field  Exercise  fSimFX)  Tool  Questionnaire 

Print  your  full  name _ 

Please  note:  Your  name  and  your  responses  to  this  questionnaire  will  never  be  made  public. 
ARI  might  use  your  name  if  it  wishes  to  contact  you  to  follow-up  on  your  experiences  with  SimFX. 

Based  on  your  admittedly  short  experience  with  SimFX  and  on  your  experience  as  a  trainer  in  the 
Infantry,  respond  to  the  questions  in  Parts  1  and  2  of  this  questionnaire  by  marking  an  "X"  in  the 
appropriate  box  of  the  7-point  scale.  Please  consider  the  entire  7-point  scale  before  making  your 
responses. 

Part  1.  Overall  Potential  Training  Value  of  SimFX _ 


1.  Would  using  SimFX  provide  a  Soldier  opportunity  to  practice  making  sound  tactical  decisions? 


NO  SOME  GREAT 

OPPORTUNITY  OPPORTUNITY  OPPORTUNITY 

2.  Would  training  with  SimFX  improve  a  Soldier’s  ability  to  make  more  rapid  tactical  decisions? 


WILL  NOT  MAY  WILL  DEFINITELY 

IMPROVE  IMPROVE  IMPROVE 

3.  Would  training  with  SimFX  make  a  Soldier  more  confident  in  his  ability  to  make  tactical  decisions? 


NOT  MORE  SOMEWHAT  MORE  MUCH  MORE 

CONFIDENT  CONFIDENT  CONFIDENT 

4.  How  challenging  is  the  overall  experience  provided  by  training  with  SimFX? 


NOT  MODERATELY  VERY 

CHALLENGING  CHALLENGING  CHALLENGING 

5.  Would  training  with  SimFX  have  a  valuable  impact  on  small  unit  leaders? 


NO  SOME  GREAT 

VALUE  VALUE  VALUE 

6.  Would  training  with  SimFX  help  small  unit  leaders  focus  on  critical  factors  that  influence  tactical 
decisions? 


NO  SOME  EXCELLENT 

FOCUS  FOCUS  FOCUS 

7.  To  what  extent  could  SimFX  teach  the  Soldier  something  new  about  decision  making  that 
is  not  now  or  not  easily  covered  in  normal  classroom  or  field  training? 


NOT  AT  ALL  SOMEWHAT  VERY  MUCH 


Continue  on  the  next  page 
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8.  To  what  extent  will  training  with  SimFX  help  a  Soldier  to  make  sound  tactical  decisions? 


NOT  AT  ALL  SOMEWHAT  VERY  MUCH 

9.  To  what  extent  will  training  with  SimFX  permit  a  Soldier  to  practice  the  types  of  decisions  he  must 
make  as  a  small  unit  leader? 


NOT  AT  ALL  SOMEWHAT  VERY  MUCH 

10.  Would  training  with  SimFX  be  a  valuable  learning  experience? 


NO  SOME  GREAT 

VALUE  VALUE  VALUE 

11.  As  a  trainer,  how  satisfied  were  you  with  the  training  opportunities  provided  by  SimFX? 


NOT  SOMEWHAT  COMPLETELY 

SATISFIED  SATISFIED  SATISFIED 


Part  2.  User  Involvement  or  Immersion  in  SimFX 


1 .  How  captivated  or  drawn  in  would  a  Soldier  be  by  the  tactical  events  and  actions  presented  in 
SimFX? 


NOT  AT  ALL  SOMEWHAT  COMPLETELY 

2.  How  often  might  a  Soldier  training  with  SimFX  be  completely  focused  on  the  decision-making  task? 


NONE  OCCASIONALLY  FREQUENTLY 

3.  How  involved  were  you  with  tactical  events  and  actions  portrayed  in  SimFX? 

I _ I _ I _ I _ I _ I _ I _ I 

NOT  AT  ALL  SOMEWHAT  COMPLETELY 

4.  How  involved  were  you  in  the  decision-making  experiences  provided  by  SimFX? 

I _ I _ I _ I _ I _ I _ I _ I 

NOT  MILDLY  COMPLETELY 

INVOLVED  INVOLVED  ENGROSSED 

5.  Were  you  so  involved  while  using  SimFX  that  you  lost  track  of  time? 

I _ I _ I _ I _ I _ I _ I _ I 

NOT  AT  ALL  SOMEWHAT  COMPLETELY 


Continue  on  the  next  page 
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Part  3.  The  Degree  to  Which  SimFX  is  Easy  or  Difficult  to  Use _ 

In  Part  3  of  the  questionnaire  draw  a  circle  around  the  letter  that  best  indicates  the  extent  to  which  you 
agree  or  disagree  with  each  statement.  Write  the  letters  NA  to  the  left  of  the  statement  number  to 
indicate  that  you  have  no  basis  for  having  an  opinion  about  the  statement. 

Strongly  disagree 
Disagree  \ 
Neither  agree  nor  disagree  |  j 

Agree  |  1  j 

Strongly  agree  I  I  |  I 


Section  A:  Using  SimFX  to  Train  Decision-Making  Skills  I  I  I  I  I 

1 .  If  it  were  available,  I  would  use  SimFX  as  a  training  tool . A  B  C  D  E 

2.  The  steps  required  to  use  SimFX  for  training  purposes  are  easy  to  perform . A  B  C  D  E 

3.  The  abbreviated  operations  order  provided  to  the  user  at  the  beginning 

of  a  scenario  were  adequate  to  understand  the  training  mission . A  B  C  D  E 

4.  The  description  of  the  factors  that  defined  each  decision  point  were 

adequate  for  the  purpose  of  the  training  exercise . A  B  C  D  E 

5.  The  types  of  decision-making  tasks  possible  with  SimFX  are  similar 

to  those  encountered  in  an  operational  environment . A  B  C  D  E 

6.  It  was  easy  to  request  and  obtain  information  while  executing  the  mission . A  B  C  D  E 

7.  The  reporting  process  was  adequate  to  keep  my  CO  informed . A  B  C  D  E 

8.  The  signals  used  on  the  SimFX  screen  for  alerting  the  user  that  a  source 

of  information  is  available  were  easy  to  understand . A  B  C  D  E 

..9.  The  printed  text-based  information  available  to  the  user  of  SimFX 

were  easy  to  understand . A  B  C  D  E 

10.  The  printed  text-based  information  available  to  the  user  of  SimFX  were 

useful  for  selecting  a  course  of  action  at  the  decision  point . A  B  C  D  E 

1 1 .  The  graphic-  or  photographic  image-based  information  available  to  the  user 

of  SimFX  were  easy  to  understand . A  B  C  D  E 

12.  The  graphic-  or  photographic  image-based  information  available  to  the  user 

of  SimFX  were  useful  for  selecting  a  course  of  action  at  the  decision  point . A  B  C  D  E 

13.  The  time  pressure  SimFX  imposes  on  the  decision  maker  is  appropriate . ...A  B  C  D  E 

14.  The  planning  map  provided  an  appropriate  amount  of  detail . A  B  C  D  E 

15.  The  “clock”  shown  in  the  SimFX  display  to  indicate  how  much  time  remains 

to  complete  a  mission  time  was  easy  to  understand . A  B  C  D  E 

Continue  on  the  next  page 
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Strongly  disagree 
Disagree  \ 
Neither  agree  nor  disagree  |  j 

Agree  |  |  | 

Strongly  agree  |  |  |  | 

I 

Section  A;  Using  SimFX  to  train  decision-making  skills  (continued)  | 

16.  The  alternative  courses  of  action  provided  with  each  decision  point  are 

similar  to  those  likely  to  arise  in  an  operational  environment . A 

17.  SimFX  can  be  used  to  train  Soldiers  about  the  proper  use  of  robotic  sensors . A 

18.  The  feedback  provided  following  a  decision  was  appropriate  for  the  purpose, 

of  training  decision-making  skills . A 

19.  If  the  SimFX  tool  were  available  I  would  use  it  to  train  my  Soldiers . A 


B  C  D  E 
B  C  D  E 

B  C  D  E 
B  C  D  E 


Section  B.  Using  SimFX  to  modifv/create  mission  scenarios  and  practice  exercises 


1 .  If  it  were  available,  I  would  personally  modify  existing  SimFX  scenarios 

or  exercises . A  B  C  D  E 

2.  If  it  were  available,  I  would  personally  create  some  new  SimFX  scenarios 

or  exercises . A  B  C  D  E 

3.  The  procedures  for  modifying  or  creating  a  SimFX  scenario  or  exercise 

are  quite  logical . A  B  C  D  E 

4.  The  steps  required  to  modify  an  existing  SimFX  scenario  or  exercise 

are  easy  to  perform . A  B  C  D  E 

5.  The  steps  required  to  create  a  new  SimFX  scenario  or  exercise  are  easy 

to  perform . A  B  C  D  E 

6.  With  just  a  little  practice  I  could  become  very  good  at  modifying  and 

creating  SimFX  scenarios  or  exercises . A  B  C  D  E 
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Part  4.  Overall  Opinion  of  the  SimFX  for  training  the  decision-making  skills 

Section  A.  The  training  capabilities  of  SimFX _ 


1 .  What  would  be  the  chief  advantages  for  using  SimFX  as  a  tool  for  training  the  decision-making  skills  of 
small  unit  leaders?  Describe  up  to  three  positive  features  of  SimFX  for  training  that  would  encourage  you 
to  use  it  as  a  training  tool. 


2.  What  would  be  the  chief  disadvantages  for  using  SimFX  as  a  training  tool  for  improving  the  decision¬ 
making  skills  of  small  unit  leaders?  Describe  up  to  three  negative  features  of  SimFX  for  training  that 
would  prevent  you  from  using  it  as  a  training  tool. 


Section  B.  The  authoring  capabilities  of  SimFX  to  modify/create  scenarios  or  exercises 

1 .  Describe  up  to  two  positive  features  of  the  authoring  capabilities  of  SimFX. 


2.  Describe  up  to  two  negative  features  of  the  authoring  capabilities  of  SimFX. 


If  you  wish  to  be  kept  informed  about  the  final  status  of  this  research  product  please  print  your 
phone  number  and  e-mail  address  below. 

Phone  number:  _ 

Email  address:  _ 


Thank  you  for  participating  in  this  demonstration  of  SimFX  and  for  completing  this  questionnaire. \ 


ARI  POC:  Dr.  Richard  E.  Christ 
706-545-2207,  ChristR  @  benning.army.mil 
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Table  C-5.  Positive  Written  Comments  About  Using  SimFX  for  Training 
(Respondent  ID) 


Simple  to  use  (5) 

Flexible  (11) 

Easy  to  understand  and  use  (15) 

Easy  to  use  (19) 

Able  to  make  up  your  own  scenarios  (19) 

Easy  to  adjust  scenarios  (5) 

This  system  appears  to  be  unlimited  in  its  capabilities  as  a  training  tool.  The  only  limitation 
would  be  the  trainer  and  his  scenarios.  (12) 

Can  be  developed  by  the  trainer  to  suit  his  situation  (14) 

The  scenarios  can  easily  be  modified  to  teach  a  wide  variety  of  decision  making  skills.  (17) 
Visual  aid  to  tactical  exercise  (11) 

Graphics  (19) 

Expand  data  base  of  exercises  in  unit  (11) 

Making  good  decisions  (19) 

Allows  the  exercise  to  focus  on  decision  making  (14) 

Requires  quick  sound  decisions  (1) 

Requires  quick  decision  with  limited  info  (3) 

The  time  limits  make  them  form  decisions  quickly,  as  well  as  analyze  info,  quickly  (10) 
Allows  PLs  to  practice  making  decisions  in  a  time  constrained  environment  (9) 

Allows  Jr.  leaders  to  practice  skills  involved  in  decision  making  such  as  map  reading  and 
graphic  control  measures  (9) 

It  makes  Soldiers  form  decisions  with  limited  information  in  a  safe  environment  (10) 

Great  way  for  leader  to  learn  how  not  to  make  hasty  decision  (think  first)  (18) 

Post  H-hour  decision  making  skills  are  hard  to  replicate  for  Co  commanders. 

This  is  an  option.  (8) 

Enables  one  to  actually  execute  as  opposed  to  simply  planning  an  operation  (7) 

Teaches  you  to  utilize  assets  (1) 

Build  confidence  not  only  in  the  Soldier  but  confidence  on  using  technology  (5) 

Gets  reader  to  think  about  resources  available  (18) 

Forcing  Soldiers  to  use  available  assets  and  inform  higher  w ill  teach  them  to  do  so  (10) 
Reminds  you  to  communicate  with  higher  (1) 

Gives  real  world  feedback  without  spending  the  money  for  troops  or  supplies  (6) 

It  gives  feedback  to  student  on  the  spot  (4) 

Soldier  can  use  SimFX  as  an  AAR  tool  to  iron  out  SOPs  (6) 

It  takes  away  all  distractions  normally  involved  with  field  training  exercises  (6) 

The  user  is  drawn  into  the  scenario  and  is  actively  engaged  which  promotes  learning.  (17) 
Use  to  reinforce  doctrine,  tactics,  or  what  ever  you  choose  (8) 

Allows  me  to  set  up  more  realistic  responses  (3) 

Doesn’t  require  a  lot  of  computer  space  (14) 

Do  not  need  to  “train”  the  Soldier  on  the  actual  software  before  training  them 
on  the  content  (17) 
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Appendix  C.  Detailed  Results  from  the  SimFX  Questionnaire 

Table  C-6.  Negative  Written  Comments  About  Using  SimFX  for  Training 
(Respondent  ID) 


More  to  fighting  than  robotic  sensors  (7) 

It  may  put  the  platoon  in  a  failed  situation  because  the  PL  did  not  check  an  asset 
which  is  not  realistic.  (10) 

Many  of  the  assets  are  not  avai!ab!e_at  the  platoon  level  and  might  confuse  things  (10) 
Automation  usage  (11) 

Time  required  to  develop  scenario  (14) 

It  takes  some  time  for  instructors  to  develop  the  scenarios  (17) 

The  scenarios  need  to  be  doctrinally  correct  in  order  to  not  confuse  junior  leaders  or 
teach  them  the  wrong  information  (9) 

Scenarios  must  be  carefully  designed  and  reviewed  to  insure  that  no  negative  actions 
are  reinforced  -  especially  actions  that  are  peripheral  to  the  main  objective  (15) 

It  takes  engaged  instructors  to  provide  additional  feedback  and  teaching  points  to  the 
students  (17) 

Need  to  develop  new  icons  for  conventional  units  (14) 

To  design  and  input  a  Co-level  activity  with  appropriate  options,  graphics,  etc.  would  be 
a  major  task  (8) 

The  need  for  computers  to  get  everyone  training  on  the  same  day  (1) 

Lack  of  computers  (3) 

Soldiers  are  sometimes  skeptical  about  using  simulations  to  train.  (They  would  rather 
train  in  the  field.)  (17) 

Nothing  fully  replaces  “boots  on  the  ground”  (1) 

Sometimes  there  is  no  correct  answer,  not  everything  you  do  on  the  battlefield  is  black 
and  white  (3) 

A  student  is  relying  on  the  SimFX  answer  (4) 

Mission  change/  Programmed  response  (11) 

From  what  I  experienced,  it  doesn’t  allow  for  different  types  of  platoons,  such  as 
mechanized,  lights,  or  mortar,  etc.  (10) 

Limited  storyline  (11) 

As  you  make  decisions  it  should  show  visual  progress  on  the  map  (19) 

Overlays  should  stay  up  at  all  times  (6) 

You  have  to  start  over  with  every  fatal  decision  -should  restart  from  there  (6) 

Viewing  in  the  1st  person  as  opposed  to  icons  to  on  a  map  is  more  effective  (7) 

No  stress  involved  (2) 

No  overall  statement  of  strengths  or  weaknesses  of  decision  making  skills  (6) 

Some  trainers  (SMEs)  do  not  fully  understand  the  nature  of  their  own  expertise  (14) 
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Appendix  C.  Detailed  Results  from  the  SimFX  Questionnaire 

Table  C-7.  Positive  Written  Comments  About  Using  SimFX  for  Authoring 
(Respondent  ID) 

Easy  system  to  understand.  User  friendly  (8) 

Simple  (2) 

Looks  simple  (11) 

Looks  flexible  (11) 

Seems  to  be  simple  (12) 

Very  easy  to  use!  (17) 

Able  to  change  the  icons  (19) 

Ability  to  use  desired  terrain  (7) 

The  ability  to  add  images,  photos,  etc.  (1) 

It  allows  the  instructor  to  change  the  scenario  to  include  assets  or  delete  assets.  This  feature 
can  make  SimFX  more  applicable  to  the  current  force  capabilities  (10) 

Point  and  click  (15) 

Word  worksheet  to  develop  initial  scenario  (14) 

Branching  chart  to  visualize  learning  path  (14) 

A  unit  can  create  scenarios  tailored  to  its  unit  mission  (1) 

Very  flexible  -  can  create  a  wide  array  of  scenarios  (17) 

You  can  tailor  make  it  to  your  mission  (6) 

Set  up  how  you  want  (3) 

Create  training  specific  to  your  unit  (3) 

Able  to  change  with  little  effort  (4) 

Work  well  with  every  changing  COE  (4) 

It  evolves  as  the  mission  continues  (6) 

This  would  be  great  for  QDXs  for  the  NCOA  schools  for  tactics  (5) 

I  think  the  system  would  allow  you  to  take  current  TDEs  and  convert  them  to  SimFX  (9) 
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Appendix  C.  Detailed  Results  from  the  SimFX  Questionnaire 

Table  C-8.  Negative  Written  Comments  About  Using  SimFX  for  Authoring 
(Respondent  ID) 


It  will  take  a  little  time  to  become  familiar  with  the  software  (17) 

To  design  and  input  a  Co-level  activity  with  appropriate  options,  graphics,  etc.  would  be 
a  major  task  (8)  [same  comment  made  with  respect  to  SimFX  training  value] 

Would  tale  a  long  time  to  truly  depict  reality  (7) 

It  puts  more  pressure  on  the  instructor  to  develop  new  scenarios  or  modification 
to  the  old  ones  (10) 

Different  instructors  may  modify  it  differently,  so  there  is  not  one  standard  in  a  company  or 
battalion  (10) 

On  the  current  battlefield  we  set  battle  updates  as  mission  changes.  I  think  some 
scenarios  that  should  happen  because  it  will  adjust  the  thinking  process  of  the  Soldiers  (6) 

Depending  on  author’s  perception  can  force  students  on  what  to  think,  not  how  to  think  (7) 

Enough  features  to  create  scenarios  for  the  current  force  to  use?  (It  was  hard  to  know 
this  from  the  demo.) 

A  good  addition  might  be  short  audio  input  for  voice  messages 

Programmed  enemy  actions 

%  of  occurrence 

Software  should  keep  the  overlay  displayed _ 
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